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(57) Abstract: An apparatus and method for generating a synthetic face image based on shape information about an input face image 
are provided. The apparatus extracts shape information, which is represented by a deformation field with respect to a predetermined 
reference image, and texture information, which is color or brightness information about an input image mapped on the reference 
image, about the input face image from face image information transmitted from a user interfacing device, and transforms a variety 
of face images, which are stored in an image database in advance and have the same shape as that of the reference image, using the 
shape information about the input face image according to a user control command, thereby synthesizing face images reflecting the 
shape information about the input face image. Accordingly, natural new images reflecting the shape of the input face image and 
having high quality regardless of the state of the input face image can be synthesized by using images, which have the same shape 
information as that about the reference image but have different types of texture information, and the extracted shape information 
about the input face image. 
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APPARATUS AND METHOD FOR GENERATING SYNTHETIC FACE IMAGE BASED ON SHAPE 

INFORMATION ABOUT FACE IMAGE 

Technical Field 

5 The present invention relates to an apparatus and method for 

generating a synthetic face image, and more particularly, to an apparatus 
and method for generating a synthetic face image based on shape 
information about an input face image. 

A face image has generally been used as a medium of showing 

10 personal character most fully and allowing easy and natural 
communication. Such a face image is used in applications such as exit 
and entry control/security systems, criminal searching/montage 
composing systems, computer interfacing, animations, and games. 
Representative applications using technology of synthesizing face 

is images are generation of a character image and makeup design. 

A caricature of a face image, which is a sort of a character image, 
is produced based on the features of a person's face. Accordingly, a 
caricature of a face image is not only used for producing animations or 
entertaining programs but also used as a symbol or icon representing a 

20 person, as a unique signature in PC communications or E-mail, and as 
an avatar of a user in the virtual reality. 

Background Art 

In order to generate caricatures, conventionally, professional 
25 painters manually draw caricatures, or face images are automatically 
processed using digital filters. Here, a method of processing an image 
using digital filters gives an input image an effect of a caricature 
manually created as a whole by realizing effects of watercolor painting or 
charcoal drawing using combinations of filters having appropriate effects 
30 on an input image. 
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When a professional painter draws a caricature manually, a 
natural and highly perfect caricature can be obtained, but it takes a long 
time and it is difficult to maintain consistent quality. Accordingly, this 
method is restricted to particular conditions. An ifaage processing 

5 technique using digital filters can be used for images picked up in an 
environment in which lighting and backgrounds are limited, so for simple 
two-dimensional images in which an object is not discriminated from a 
background, the quality of output images is depending on lighting or 
other changes in an environment. No proper methods for supplementing 

io this technique have been proposed. Moreover, since shape information 
about an object is not separately generated in a conventional caricature 
generation method, it is very complex to perform a revising operation 
such as exaggerating of the features of the face in a generated 
caricature or changing of a facial expression, and it is nearly impossible 

15 to restore a face image or extend the generated caricature to a 
three-dimensional avatar. 

In the case of makeup design, conventionally, a consumer 
determines a style indirectly based on pictures of models having on 
makeup. Recently, a makeup designing method using a computer has 

20 been introduced. This method allows to a user to apply products to 
sample model images in various ways but does not provide natural 
effects that can be obtained when directly applying makeup to the face 
image of a consumer. In other words, even if the same makeup product 
is used, color can be different depending on various and complex 

25 conditions such as shades and reflected light due to illumination and 
features of the shape of the face, so it is nearly impossible to naturally 
derive the makeup effects on the face image of a consumer from the 
makeup effects on a sample model image. 

30 Disclosure of the Invention 
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To solve the above problems, it is an object of the present 
invention to provide an apparatus and method for generating a synthetic 
face image based on shape information about a face image, in which 
shape information about a face is extracted from an input face image, 
5 and a synthetic face image is generated based on the extracted shape 
information, so that a user can obtain a natural and elaborate caricature 
image, view a makeup design image using his/her face image in advance, 
easily add a variety of accessories to the synthetic image, easily change 
the synthetic image, and view the result image of adding accessories to 
10 or changing the synthetic image in real time. 

To achieve the object of the invention, there is provided an 
apparatus • for synthesizing a new face image based on shape 
information about an input face image. The apparatus includes a user 
interfacing device for receiving face image information and a user control 
command, transmitting the same to an image processing device, 
receiving synthesized face image information from the image processing 
device, and outputting or storing the synthesized face image information 
according to the user control command; and an image processing device 
for extracting shape information, which is represented by a deformation 
field with respect to a predetermined reference image, and texture 
information, which is color or brightness information about an input 
image mapped on the reference image, about an input face image from 
face image information transmitted from the user interfacing device, and 
for transforming a texture image selected from among texture images, 
25 which are stored in an image database in advance and have the same 
shape as that of the reference image, or an image generated by 
performing a weighted summation of the selected texture image and a 
texture image reflecting the extracted texture information, using the 
shape information about the input face image according to the user 
30 control command, thereby generating a synthetic face image. 
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There is also provided a method for synthesizing a new face 
image based on shape information about an input face image. The 
method includes the steps of (a) extracting shape information, which is 
represented by a deformation field with respect to shape information 

5 about a predetermined reference image, and texture information, which 
is color or brightness information about an input image mapped on the 
reference image about an input face image from input face image 
information; and (b) transforming a texture image selected from among 
texture images, which are stored in an image database in advance and 

10 have the same shape as that of the reference image, or an image 
generated by performing a weighted summation of the selected texture 
image and a texture image reflecting the extracted texture information, 
using the shape information about the input face image according to a 

v.* 

user control command, thereby generating a synthetic face image. 

15 

Brief Description of the Drawings 

FIG. 1 A is a block diagram of the functional configuration of a first 

■ 

embodiment of a synthetic face image generating apparatus according to 

the present invention; 
20 FIG. 1B is a block diagram of the functional configuration of a 

second embodiment of a synthetic face image generating apparatus 
according to the present invention; 

FIG. 2 is a block diagram of the mechanical configuration of a 
computer system in which the first and second embodiments of the 
25 present invention are implemented; 

FIG. 3 is a basic flowchart of a procedure of generating a 
synthetic face image according to the present invention; 

FIG. 4 is a detailed flowchart of the step of extracting face 
information shown in FIG. 3; 
30 FIG. 5 is a flowchart of a procedure of generating a caricature 
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image according to a face image synthesizing method performed by an 
apparatus for generating a synthetic face image according to the present 
invention; 

FIG. 6 is a flowchart of a procedure of generating a caricature 
5 image according to a sample image replacing method performed by an 
apparatus for generating a synthetic face image according to the present 
invention; and 

FIG. 7 is a flowchart of a makeup design procedure performed by 
an apparatus for generating a synthetic face image according to the 
io present invention. 

Best mode for carrying out the Invention 

Hereinafter, embodiments of the present invention will be 
described in detail with reference to the attached drawings. 

15 FIGS. 1A and 1B are block diagrams of a first embodiment 1 and 

a second embodiment 40, respectively, of a synthetic face image 
generating apparatus based on shape information about a face image 
according to the present invention. The first embodiment 1 shown in 
FIG. 1 includes one or more user interfacing devices 10a and 10b, a 

20 communication network 20, and an image processing device 30, thereby 
operating in a network environment. The second embodiment 40 
shown in FIG. 2 includes a user interfacing device 50 and an image 
processing device 60, thereby operating in a single computer system. 

The user interfacing devices 10a and 10b and the image 

25 processing device 30 of the first embodiment 1, and the second 
embodiment 40 are realized as a computer system 70 including a 
computer 72 with at least one central processing unit (CPU) 74 and a 
memory unit 73, an input device 75, and an output device 76, as shown 
in FIG. 2. The elements of the computer system 70 are connected to 

# 

30 one another through one or more bus structures 77. 

5 
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The CPU 74 includes an arithmetical and logical unit (ALU) 741 
for performing an arithmetical operation and a logical operation, a 
register set 742 for temporarily storing data and commands, and a 
control unit 743 for controlling the operations of the computer system 70. 
The CPU 74 used in the present invention is not restricted to a 
particular structure manufactured by a particular company but can be 
any type of processor having the above basic structure. 

The memory unit 73 includes a high-speed main memory 731 and 
an auxiliary memory 732 for storing data for a long time. The main 
memory 731 may include a random access memory (RAM) 
semiconductor chip and a read only memory (ROM) semiconductor chip. 
The auxiliary memory 732 may include a floppy disc, a hard disc, a 
CD-ROM, a flash memory, and a device for storing data using electricity, 
magnetism, light or other recording medium. The main memory 731 
is may include a video display memory for displaying an image through a 
display device. It will be understood by those skilled in the field of the 
present invention that the memory unit 73 can include a variety of 
replaceable elements that have different storing performance. 

The input device 75 may include a keyboard, a mouse, and a 
20 physical converter (for example, a microphone). The output device 76 
may include a display unit, a printer, and a physical converter (for 
example, a speaker). Alternatively, a network interface, a modem, or 
the like can be used as an input/output device. 

Thecomputer system 70 is provided with an operating system and 
25 at least one application program. An operating system is a series of 
software controlling the operations of the computer system 70 and 
assignment of resources. An application program is a series of 
software performing work, which is requested by a user, using available 
computer resources through the operating system. This software is 
30 stored in the memory unit 73. Consequently, a computer-basis 
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synthetic face image generating apparatus according to the present 
invention is realized as the combination of the computer system 70 and 
one or more application programs which are installed and operate in the 
computer system 70. 

5 The first embodiment 1 shown in FIG. 1 A has the same functions 

as the second embodiment 40 shown in FIG. 1B, with the exception that 
the embodiment 1 further includes communication processors 14 and 31 
for data transmission through the communication network 20. Thus, the 
present invention will be described on the basis of the first embodiment 

10 1. 

Referring to FIG. 1A, each of the user interfacing devices 10a and 
10b receives face image information and a user control command from a 
user, receives a synthesized image in response to the user control 
command, and revises, stores or outputs the synthesized image. Each 

15 of the user interfacing devices 10a and 10b includes an image 
information input unit 11, a user command input unit 12, an input/output 
controller 13, a communication processor 14, an image revision unit 15, 
an image storage unit 16, and an output unit 17. 

The image information input unit 11 receives face image 

20 information from a user and can be realized as, for example, a scanner 
or a digital camera. In addition, the image information input unit 1 1 may 
include a plurality of cameras for receiving images picked up at different 
angles and a camera supplementary device such as a light adjustor. 
Since the image information input unit 11 as an element of the present 

25 invention should be considered in the functional aspect, the image 
information input unit 11 should be largely construed as including not 
only the input device 75 of FIG. 2 but also the auxiliary memory 732 of 
FIG. 2 which stores face image information in advance. 

The user command input unit 12 receives user control commands 

30 (for example, user information, a face image synthesis control signal, 
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and an image revision control signal) from a user. The user command 
input unit 12 can be realized as a device such as a keyboard, a mouse or 
a touch screen through which a user can input a command or 
information. 

5 The input/output controller 13 controls face image information 

input through the image information input unit 11 and a user control 
command input through the user command input unit 12 to be 
transmitted to the image processing device 30 through the 
communication processor 14. The input/output controller 13 also 

io receives new image information which is synthesized by the image 
processing device 30 in response to the user control command and 
controls the new image information to be revised, stored or output. 

The communication processor 14 is connected to the input/output 
controller 13 and transmits data to or receives data from the image 

15 processing device 30 through the communication network 20. For 
example, the communication processor 14 can be realized as a device 
including a serial/parallel port, a universal serial bus (USB) port or an 
IEEE1394 port for internal connection, and an Ethernet card for 
transmitting and receiving data including image information through 

20 Internet. 

The image revision unit 15 is connected to the input/output 
controller 13 and revises the angle, size or texture of an image 
corresponding to image information synthesized by and transmitted from 
the image processing device 30 in response to a user control command 

25 input through the user command input unit 1 2. 

The image storage unit 16 corresponds to the auxiliary memory 
732 of FIG. 2 and stores new image information synthesized by and 
transmitted from the image processing device 30 or image information 
revised by the image revision unit 15 under the control of the input/output 

30 controller 13. 
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The output unit 17 corresponds to the output device 76 of FIG. 2. 
Under the control of the input/output controller, the output unit 17 
displays user interface display information used for receiving a user 
control command required when the image processing device 30 

5 synthesizes a new image and displays or prints new image information 
synthesized by and transmitted from the image processing device 30 or 
image information revised by the image revision unit 15. 

The communication network 20 transmits data between one or 
more user interfacing devices 10a and 10b and the image processing 

10 device 30 in the first embodiment 1 of the present invention shown in FIG. 
1A. The communication network 20 can be realized as one of a variety 
of types of networks such as wire/wireless Internet, local area networks 

and private lines. 

In the first embodiment 1 of the present invention shown in FIG. 1, 

15 the image processing device 30 processes image information transmitted 
from one or more user interfacing devices 10a and 10b, and synthesizes 
a new image based on the image information in response to a user 
control command, and transmits the synthesized image to a 
corresponding user interfacing device. The image processing device 30 

20 includes a communication processor 31 , an image processor 32, and an 

image database (DB) 33. 

The communication processor 31 transmits data to and receives 
data from, one or more user interfacing devices 10a and 10b. 
Corresponding to the communication processor 14 in each of the user 
25 interfacing devices 10a and 10b, the communication processor 31 can 
be realized as a device including an Ethernet card transmitting and 
receiving data including image information through Internet and a 
serial/parallel port, a universal serial bus (USB) port or an IEEE1394 port 

for internal connection. 
30 The image processor 32 extracts shape information about an 



9 



» 



WO 02/17234 PCT/KROl/01167 



10 



input face image, indicated as a deformation field with respect to a 
reference image, and texture information, i.e., color or brightness 
information, about the input face image mapped on the reference image 
from face image information transmitted from the user interfacing 
devices 10a and 10b, analyzes a user control command transmitted from 
the user interfacing devices 10a and 10b to interpret a user's request, 
and synthesizes a new face image using the extracted shape information, 
the extracted texture information and a variety of images stored in the 
image DB 33 according to the interpreted user's request. The image 
processor 32 includes a face information extractor 321, a face image 
synthesizer 322, a partial image replacing unit 323, and an accessory 

image adding unit 324. 

The face information extractor 321 extracts from face image 
information transmitted from the user interfacing devices 10a and 10b 
i shape information about an input face image, indicated as a deformation 
field with respect to a reference image, and texture information, i.e., color 
or brightness information, of the input face image mapped on the 
reference image using the shape information. 

The face image synthesizer 322 synthesizes a new face image by 
o transforming a texture image selected from among texture images stored 
in the image DB 33 in response to a user control command or 
transforming an image, which is generated by performing a weighted 
summation of the selected texture image and a texture image reflecting 
texture information extracted by the face information extractor 321, using 
> 5 shape information about an input image, which is extracted by the face 

information extractor 321. 

The partial image replacing unit 323 replaces a portion or the 
entire area of a new face image synthesized by the face image 
synthesizer 322 with a sample image having the highest similarity among 
30 sample images stored in the image DB 33. 
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The accessory image adding unit 324 adds an accessory image, 
which is selected from among accessory images stored in the image DB 
33 according to a user control command, to a face image synthesized by 
the face image synthesizer 322. 

The image DB 33 previously stores image information necessary 
for processing an input face image in the image processor 32. The 
image DB 33 includes a face model DB 331 , an additional image DB 332, 
a sample image DB 333, a makeup image DB 334, and an accessory 
image DB 335. 

The face model DB 331 stores various types of information (a 
shape average, a texture average, shape eigenvectors, and texture 
eigenvectors which are previously obtained with respect to a plurality of 
model face images) used for extracting shape information and texture 
information, based on a reference image, from an input face image in the 
face information extractor 321. The various types of information stored 
in the face model DB 331 will be described in detail later when FIG. 4 is 
described. 

The additional image DB 332 stores information about caricature 
images which have the same shape as that of a reference image but 
have different styles of texture information such as an animation style, a 
sketch style and a watercolor printing style. 

The sample image DB 333 stores information about various 
caricature sample images including changes in a shape and a facial 
expression at each particular portion of a face image. 

The makeup image DB 334 stores information about makeup 
design images which have the same shape as that of a reference image 
and are expressed with texture information expressing a variety of 

sample makeups. 

The accessory image DB 335 stores information about images 

such as glasses, hairstyles, hats, earrings, and bodies which can be 
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added to a synthesized face image. 

,„ melt embodiment 1 o, .he present invention shown ,n RGL 

1A one or more user interfacing devious 10a and 10b are connected «o 
1A, one or moi communication 
a single image processing device 30 tnro g AKamativeW 
processors 14 and 31 and the communication network 20. AKematve*. 
asTme second embodiment 40 of the present invention shown i , FML 
Tb the user interfacing device SO and the image processrng de«ce 60 

„„ mreorated and operated within a single computer system 70. 
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used for synthesizing a face image, different new images having the 
shape of the input face image can be synthesized. 

The synthesized face image is transmitted to the user interfacing 
device 10a, 10b or 50 and displayed through the output unit 17 or 57. 
The user command input unit 12 or 52 of the user interfacing device 10a, 
1 0b or 50 receives a user control command indicating whether to change 
the shape information about the displayed face image from the user in 
step S1 3. 

When the user control command instructs to change the shape 
information, the shape information about the input face image is changed 
according to the user control command (for example, a control signal 
instructing partial transform such as enlarging or reducing a particular 
portion of the displayed face image by dragging the portion with a mouse 
or a control signal instructing entire transform such as exaggerating the 
entire face using a slide bar), and the operation goes back to step S12 to 
synthesize a new face image. 

When the user does not want to change the shape information in 
step S13, in response to an additional user control command, the 
accessory image adding unit 324 or 624 adds various accessory images 
stored in the image DB 33 or 63 to the face image synthesized in step 
S12, or the partial image replacing unit 323 or 623 replaces a particular 
portion of the face image synthesized in step S12 with various sample 
images stored in the image DB 33 or 63, thereby adding various 
additional effects to the synthesized face image in step S14. 

Thereafter, the face image synthesized by the image processing 
device 30 or 60 is transmitted to the user interfacing device 10a, 10b or 
50 and displayed for the user. The image revision unit 15 or 55 finally 
revises the synthesized face image in response to a user control 
command (an image revision control signal) received through the user 
command input unit 12 or 52 in step S15. The synthesized face image 
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revised by the image revision unit 15 or 55 is stored in the image storage 
unit 16 or 56. or displayed or panted by the output unit 17 or 57 ,n step 

S16 ' The faoe information extracting step S11 of FIG. 3 can be 
summarized as a procedure of obtaining shape information S. and 
texture information r. tan an input face image based on a face model. 

In the present invention, shape information about a face image ,s 
expressed by a deformation field with respect to a reference ,mage, and 
texture information about a faoe image is expressed by color or 
brightness information about an input image mapped on the reference 
image, in other words, shape information S about a face image . 
defined as the positional difference between each point „ on a reference 

im age (( =1 n; n is the number of predetermined points . on the 

reference image) and each corresponding point on the face .mage^ 
Texture information T about a face image is defined as ate color or 
brightneas value of each point on an input image with respect £ each 

corresponding point » 0=1 ") on a reference ,mage. WMe a 

synthase image obtained using a shape average and a texture average 
is used as a reference image in an embodiment of the present .nvenaon 
, a reference image which can be used in the present invention . not 
restricted to the above embodiment. Any one among . face images 
prepared previously can be used as a reference image. 

FaL models stored in the faoe mode, DB 331 or 631 are 
previously obtained according to the following procedure. First, shape 

, 5 information « 0=1 «0 and texture information 2) 0=1 -) 

' extracted from the previously prepared m face images based on a 
reference image. Thereafter, a shape average S composed o 

averages of shape information S, 0=1 «> * each point p, 0=1 ») 

and a texture average f composed of averages of texture information 
j_ .... m ) at each point „ 0=1 "> are ° btained - Then * 
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covanance C s of a shape difference S y =S,-S <M. - *> and a 
covariance O of a texture difference f^T r T 0*1. .... m) are 
obtained. 

These obtained values are processed by principal component 

analysis, thereby obtaining the shape eigenvector s, (i=1 m-1) and 

texture eigenvector /, (/=1, .... m-1) of a covariance of the m face models. 
Therefore, a face image can be expressed by Equation (1) based on the 
shape eigenvector * (i=1 m-1) and the texture eigenvector t, (M 

m-1). 
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(Here. & £e fl" -1 . and m is the number of face models) 
Through such a procedure, the shape average S , the texture 

average f , the shape eigenvector * (z=1 m-1), and texture 

eigenvector n (/=1, .... m-1) are stored in the face model DB 331 or 631 
and used for extracting shape information and texture information about 

an input face image. 

The face information extracting step S11 of FIG. 3 will be 
described in detail with reference to FIG. 4. In step S1 1 1 of FIG. 4,. the 
input face - image is normalized. More specifically, predetermined 
feature points (for example, a central point of each pupil and a central 
point of a lip) are extracted from an input face image, and then the input 
face image is moved up, down, to the left and to the right and the size 
thereof is adjusted so that the positions of the extracted feature points of 
the input face image can be located at the corresponding feature points 
of a reference image. Such an image normalizing step can be 
automatically performed through predetermined software or can be 
manually performed in response to a control command input by the user. 
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Since a detailed procedure thereof is beyond the scope of the present 
invention, a description thereof will be omitted. 

In step S112, shape information is estimated. More specifically, 
a hierarchical, gradient-based optical flow algorithm (Lucas and Kanade) 
5 is applied to the normalized input face image obtained in step S111 and 
to the reference image (or a synthetic texture estimation image T m j. 
having the same shape as that of the reference image), thereby 
estimating shape information Stnjnj (the value of the positional 
difference between points on the normalized input face image and 
10 corresponding points on the reference image) based on the reference 
image. By using the hierarchical, gradient-based optical flow algorithm, 
the correspondence between two similar images can be expressed by an 
optical flow using the intensity of the two images. 

The shape information Smj*j obtained according to a hierarchical, 
15 gradient-based optical flow algorithm in step S112 may include an error 
value due to a light or shadow in the input face image. Accordingly, in 
step S113, compensation is performed on the shape information Sm _prej- 
More specifically, the error value is compensated for by sequentially 
performing linear decomposition based on the shape eigenvector s f 
20 (i=1, m-1) and linear superposition with respect to the shape 
information S mjn j t thereby obtaining the error compensated shape 
information S in _ cor rectj- Here, to increase the degree of freedom in 
transform, it is preferable to use as the result value of the shape 
information a weighted sum S inJ of the estimated shape information 
25 Smjntj obtained in step S112 and the compensated shape information 
S in _correctj obtained in step S113, as calculated by Equation (2). 

(Here, 0<FF<1) 

30 In step S114, backward warping is performed. More specifically, 
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the input face image is transformed into the reference image using the 
shape information S u ,j obtained in step S113. This procedure is 
referred to as "backward warping". 

In step S115 t linear decomposition is performed on texture 

5 information about the backward warped image based on the texture 
eigenvector t f (z=1, m-1), and then linear superposition is performed 
on the result of linear decomposition, thereby obtaining texture 
information T in j about the input face image. 

Thereafter, the normalized input face image obtained in step S1 12 

io is replaced with the input face image transformed into the reference 
image in step S114, and the reference image is replaced with a texture 
image having the same shape as that of the reference image. Then, 
steps S112 through S116 are repeated, thereby obtaining S^j (£=1, ...)■ 
In other words, in a k-Xh repetition, S in j, S inj3 rej t and S in __correctj in steps 

is S1 12 and S1 13 are replaced with S inJ[i S if ,_pre_k, and S in _correctj, respectively, 
and S in j in step S114 is replaced with S in ( zz S in j+„.+ Si„ k ). In addition, 
Tin j in step S115 is replaced with T inJ( . T inJ obtained in the last 
repetition becomes the final texture information about the input face 

image. Such a repetition is performed until the vector norm \s in k \ of 

20 Stn j. is less than a predetermined threshold value or until the number of 
repetitions is equal to or greater than a predetermined number, thereby 
obtaining shape information S in about the input face image based on the 
reference image in step S1 17. 

Thereafter, the input face image can be restored using the shape 

25 information S in and the texture information T in about the input face image, 
which is obtained based on the reference image. In other words, the 
texture information 7^ about the input face image obtained based on the 
reference image is transformed according to the shape information S in 
about the input face image obtained based on the reference image, 

30 thereby synthesizing the input face image. 
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Representative examples in which the characteristics of such a 
synthetic image can be utilized are generation of a caricature of a face 
image and makeup design. A method of generating a caricature of a 
face image is divided into a face image synthesizing method and a 

5 sample image replacement method. 

A procedure of generating a caricature image using the face 
image synthesizing method performed in the synthetic face image 
generating apparatuses 1 and 40 based on the shape information about 
a face image according to the present invention will be described with 

10 reference to FIG. 5. 

The face information extractor 321 or 621 of the image processor 
32 or 62 receives an input face image from the user interfacing device 
10a, 10b or 50 in step S20, and extracts shape information S in and 
texture information T in about the input face image based on a 

15 predetermined reference image in step S21 . 

Thereafter, the face image synthesizer 322 or 662 presents 
different styles of caricature images (for example, an animation style, a 
sketch style, and a watercolor printing style) stored in the additional 
image DB 332 or 632 to a user through the user interfacing device 10a, 

20 10b or 50 so that the user can select a desired style of caricature in step 
S22. Here, different styles of caricature images stored in the additional 
image DB 332 or 632 have the same shape as that of the reference 
image. 

In step S23, the face image synthesizer 322 or 622 synthesizes 
25 the selected style of caricature image or an image, which is generated by 
performing a weighted summation of the selected caricature image and 
an image reflecting the texture information T,„ about the input face image, 
with the shape information S in about the input face image, thereby 
generating a caricature image reflecting the shape information about the 
30 user's face. 
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In step S24, the synthesized caricature image is transmitted to the 
user interfacing device 10a, 10b or 50 and displayed through the output 
unit 17 or 57, and the user command input unit 12 or 52 of the user 
interfacing device 10a, 10b or 50 receives a user control command 

5 indicating whether to change the shape information about the displayed 
caricature image from the user. 

When the user control command instructs to change the shape 
information, the shape information Sp, is changed according to the user 
control command (for example, a control signal instructing partial 

10 transform such as enlarging or reducing a particular portion of the 
displayed face image by dragging the portion with a mouse or a control 
signal instructing entire transform such as exaggerating the entire face 
using a slide bar), and the operation goes back to step S22 to synthesize 
a new caricature image. 

is in step S25, in response to an additional user control command, 

the accessory image adding unit 324 or 624 retrieves various accessory 
images (glasses, hairstyles, hats, earrings, and bodies) stored in the 
accessory image DB 335 or 635, and adds the same to the caricature 
image. When the accessory image adding unit 324 or 624 adds an 

20 accessory image, more natural result can be obtained by automatically 
performing a size and position adjustment using the shape information 
Sin extracted in step S21 . In addition, the partial image replacement unit 
323 or 623 replaces a particular portion of the caricature image with a 
sample image retrieved from the sample image DB 333 or 633, thereby 

25 expressing happiness, sadness or anger on the caricature image. 
Moreover, • a moving picture effect can be accomplished by using 
animation frames expressing changes in the face. 

Thereafter, the caricature image synthesized by the image 
processing device 30 or 60 is transmitted to the user interfacing device 

30 10a, 10b or 50 and displayed for the user. The image revision unit 15 
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or 55 finally revises the synthesized caricature image in response to a 
user control command (an image revision control signal) received 
through the user command input unit 12 or 52 in step S15. The 
synthesized caricature image revised by the image revision unit 1 5 or 55 

5 is stored in the image storage unit 16 or 56, or displayed or printed by 
the output unit 17 or 57 in step S16. 

The caricature image obtained through such a method can be 
immediately used for a particular purpose. Alternatively, the caricature 
image can be used as a draft when an artist produces a caricature, 

10 thereby increasing productivity during manual treatment. 

A procedure of generating a caricature image using the sample 
image replacement method performed in the synthetic face image 
generating apparatuses 1 and 40 based on the shape information about 
a face image according to the present invention will be described with 

is reference to FIG. 6. 

FIG. 6 further includes step S35 and S36 in addition to steps 
shown in FIG. 5. Thus, descriptions of the other steps S30 through S34 
and S37 through S39 will be omitted. In other words, the method of FIG. 
6 is a method of replacing a part or the entire portion of a caricature 

20 image synthesized according to a method as shown in FIG. 5 with a 
sample image prepared in the sample image DB 333 or 633. 

Sample images stored in the sample image DB 333 or 633 are 
formed based on the result of performing a statistic analysis on shape 
information about different faces. The sample image formation method 

25 can be divided into two cases: a case where transformation of a sample 
image is admitted, and a case where transformation of a sample image 
is not admitted. 

According to the sample image formation method in the case 
where transformation of a sample image is admitted, a normalized 
30 sample image is formed, and the size and shape of the sample image 
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are transformed based on shape information S in extracted from an input 
face image in step S31. This method has advantages of substantially 
reflecting the shape of the input face image and requiring a small 
number of sample images but has disadvantages of distorting sample 
5 image by transformation and deteriorating entire picture quality. 

When using the sample image formation method in the case 

m 

where transformation of a sample image is not admitted, a part or the 
entire portion of a caricature image is replaced with only a previously 
formed sample image to synthesize a new caricature image. 

10 Accordingly, this method has an advantage of accomplishing high 
definition but is disadvantageous in that it is difficult to substantially 
reflect the shape of the input image and many sample images as 
possible as can be formed should be prepared in advance. 

A method of measuring a difference D used in step S35 of FIG. 6 

15 can be expressed by Equation (3). 



n n 



D = W • £(C„ - C ri ) + 0 - W) • £ (Q - C„) . ..(3) 



/-I M 



Here, 0<*F<1, Cj n) is shape information about an input 

image, C„ (z=1, n) is shape information about a sample image, C ti 

20 (z=1, n) is the difference T re / between texture information about the 
input image and texture information about a reference image, and C qi 
(/=1, n) is the difference T ref between texture information about the 
sample image and the texture information about the reference image. 
Instead of immediately applying shape information and texture 

25 information to C sh C r/ , C ih and C qh coefficients of eigenvectors obtained 
by linearly analyzing the shape information and the texture information, 
as shown in Equation (1), can be used. Here, each coefficient has a 
dimension of 

The partial image replacing unit 323 or 623 measure the 
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difference D between the input image and each sample image in step 
S35 and replaces a part or the entire portion of a caricature image with a 
sample image having a minimum difference in step S36. 

When a caricature image is generated according to the sample 

5 image replacement method, instead of the entire generated caricature 
image, the codes of a sample image with which a part or the entire 
portion of the caricature image will be replaced are compressed and 
transmitted in a low-speed communication environment, thereby 
markedly increasing a compression ratio. 

10 A procedure of designing a makeup performed in the synthetic 

face image generating apparatuses 1 and 40 based on the shape 
information about a face image according to the present invention will be 
described with reference to FIG. 7. 

FIG. 7 is the same as FIG. 5, with the exception that FIG. 7 

15 includes steps S42 through S45, instead of S22 through S24 of FIG. 5. 
Thus, descriptions of the other steps S40, S41, and S46 through S48 will 
be omitted. 

In step S42, the face image synthesizer 322 or 662 presents a 
variety of sample makeup images stored in the makeup image DB 334 or 

20 634 to a user through the user interfacing device 10a, 10b or 50 so that 
the user can select a desired sample makeup image. Here, the sample 
makeup images have the same shape as that of a reference image. 

In step S43, the face image synthesizer 322 or 622 transforms the 
selected sample makeup image or an image, which is generated by 

25 performing a weighted summation of the selected sample makeup image 
and an image reflecting texture information T in extracted from an input 
face image in step S41, using shape information extracted from the 
input face image in step S41 , thereby synthesizing a face image in which 
the selected sample makeup image is applied to the user's face image. 

30 In step S44, the synthesized face image reflecting the sample 
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makeup image is transmitted to the user interfacing device 10a, 10b or 
50 and displayed through the output unit 17 or 57, and the user 
command input unit 12 or 52 of the user interfacing device 10a, 10b or 
50 receives a user control command instructing revision of the makeup 

5 of the displayed face image from the user. The face image synthesizer 
322 or 622 revises the face image reflecting the sample makeup image 
according to the user control command, and the revised face image is 
transmitted to the user interfacing device 10a, 10b or 50 and displayed 
through the output unit 17 or 57. 

10 Thereafter, in step S45, the user command input unit 12 or 52 of 

the user interfacing device 10a, 10b or 50 receives from the user a user 
control command indicating whether the user is satisfied with the 
displayed face image. Here, when the user control command indicates 
"satisfied", the procedure proceeds to step S46 of adding an accessory. 

15 In contrast, when the user control command indicates "not satisfied", the 
procedure returns to step S42. 

While this invention has been particularly shown and described 
with reference to preferred embodiments thereof, it will be understood by 
those skilled in the art that various changes in forms and details may be 

20 made therein without departing from the spirit and scope of the invention. 
The above embodiments have been used in a descriptive sense only and 
not for purpose of limitation. Therefore, the scope of the invention will 
be defined by the appended claims not by the above description, and it 
should be construed that modifications made within the scope of the 

25 invention are covered by the present invention. 

Industrial Applicability 

According to the present invention, first, shape information, which 
is represented by a deformation field with respect to a reference image, 
30 about an input face image is extracted from the input face image, and 
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images which have the same shape as that of the reference image but 
have different types of texture information and the extracted shape 
information about the input face image are used, thereby synthesizing 
natural new images reflecting the shape of the input face image and 
5 having high quality regardless of the state of the input face image. 
Accordingly, the present invention can be effectively utilized in a variety 
of fields such as generation of character images, virtual makeup design, 
making of montages for criminal search, animations and games. 

Second, in case of generating caricature images, different 

10 caricatures including the characteristics of the shape of a user can be 
immediately generated, and a part or the entire portion of a generated 
caricature can be exaggerated or transformed. In addition, by using 
information about the shape of a user's face, a complex image revision 
process can be simplified and automatized, . thereby improving 

15 productivity in generating characters. 

Third, in case of makeup design, a user can easily create and 
check a makeup design on the user's face image and can easily revise 
the makeup design partially or entirely. 

Fourth, the result of adding a variety of accessories to a new 

20 synthesized face image can be immediately checked. In addition, the 
present invention can be easily applied to a variety of applications such 
as realization of an avatar in the virtual reality based on shape 
information, restoration of a three-dimensional face image, and video 
chatting which need a face image. 
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What is claimed is: 

1. An apparatus for synthesizing a new face image based on 
shape information about an input face image, the apparatus comprising: 

a user interfacing device for receiving face image information and 

5 a user control command, transmitting the same to an image processing 
device, receiving synthesized face image information from the image 
processing device, and outputting or storing the synthesized face image 
information according to the user control command; and 

an image processing device for extracting shape information and 

10 texture information about an input face image from face image 
information transmitted from the user interfacing device, the shape 
information being represented by a deformation field with respect to a 
predetermined reference image, the texture information being color or 
brightness information about an input image mapped on the reference 

15 image, and for transforming a texture image selected from among texture 
images, which are stored in an image database in advance and have the 
same shape as that of the reference image, or an image generated by 
performing a weighted summation of the selected texture image and a 
texture image reflecting the extracted texture information, using the 

20 shape information about the input face image according to the user 
control command, thereby generating a synthetic face image. 

* 

2. The apparatus of claim 1, wherein the user interfacing 
device and the image processing device are realized within a single 

25 computer system. 

3. The apparatus of claim 1, wherein the user interfacing 
device and the image processing device are separately realized in two or 
more computer systems, respectively, the apparatus further comprising a 

30 communication network for transmitting data and receiving data between 

25 
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the user interfacing device and the image processing device. 

4. The apparatus of claim 2 or 3, wherein the image 
processing device comprises: 

5 a face information extractor for extracting the shape information 

and the texture information about the input face image from the face 
image information transmitted from the user interfacing device, the 
shape information being represented by a deformation field with respect 
to the predetermined reference image, the texture information being 

10 color or brightness information about the input image mapped on the 
reference image; 

a face image synthesizer for transforming the texture image 
selected from among the texture images, which are stored in the image 
database in advance and have the same shape as that of the reference 

15 image, or the image generated by performing a weighted summation of 
the selected texture image and the texture image reflecting the extracted 
texture information, using the shape information about the input face 
image according to the user control command, thereby generating the 
synthetic face image; and 

20 an image database for storing information about the reference 

image and texture information about a variety of images having the same 
shape as that of the reference image. 

5. The apparatus of claim 4, wherein the image database 
25 comprises: 

a face model database for storing a shape average, a texture 
average, shape eigenvectors, and texture eigenvectors which are 
obtained by performing a principal component analysis on a shape 
average, a texture average, a covariance of a shape difference, and a 
30 covariance of a texture difference which are obtained from reference 

26 
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image based shape information and texture information which is 
extracted from a plurality of face images; 

an additional image database for storing information about 
different styles of caricature images having the same shape as that of 
5 the reference image; and 

a makeup image database for storing information about a variety 
of makeup design images having the same shape as that of the 
reference image. 

10 6. The apparatus of claim 5, wherein the face information 

extractor comprises: 

a normalization module for normalizing the input face image with 
respect to the reference image; 

a shape information estimation module for estimating the 
is reference image based shape information by performing a hierarchical, 
gradient-based optical flow algorithm on the normalized input face image 
and the reference image; 

a shape information compensation module compensating for for 
an error value of the shape information estimated in the shape 
20 information estimation module by performing linear decomposition and 
linear superposition on the estimated shape information based on the 
shape eigenvectors stored in the face model database; 

a backward warping module for transforming the shape of the 
input face image into the shape of the reference image using the 
25 compensated shape information; 

a texture information determining module for determining the 
texture information about the input face image by performing linear 
decomposition and linear superposition on texture information about the 
backward warped image based on the texture eigenvectors stored in the 
30 face model database; and 
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a repetition module for repeating the above modules until a 
predetermined condition is satisfied, thereby generating the shape 
information about the input face image based on the reference image. 

5 7. The apparatus of claim 5, wherein the image database 

further comprises a sample image database for storing information about 
caricature sample images having different facial expressions at each 
particular portion of a face image, and the image processing device 
further comprises a partial image replacing unit for replacing a part or the 

« 

10 entire portion of the synthetic face image generated by the face image 
synthesizer with a sample image having a highest similarity to the 
synthetic face image among the sample images stored in the sample 
image database. 

.- 

is 8. The apparatus of claim 5, wherein the image database 

further comprises an accessory image database for storing information 
about a variety of accessory images which can be added to the synthetic 
face image, and the image processing device further comprises an 
accessory image adding unit for adding an accessory image, which is 

20 selected from among the accessory images stored in the accessory 
image database according to a user control command, to the synthetic 
face image generated by the face image synthesizer. 

4 

9. A method for synthesizing a new face image based on 
25 shape information about an input face image, the method comprising the 
steps of: 

(a) extracting shape information and texture information from input 
face image, the shape information being represented by a deformation 
field with respect to a predetermined reference image, the texture 
30 information being color or brightness information about an input image 
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mapped oh the reference image; and 

(b) transforming a texture image selected from among texture 
images, which are stored in an image database in advance and have the 
same shape as that of the reference image, or an image generated by 
5 performing a weighted summation of the selected texture image and a 
texture image reflecting the extracted texture information, using the 
shape information about the input face image according to a user control 
command, thereby generating a synthetic face image. 

10 10. The method of claim 9, wherein step (a) comprises sub 

steps of: 

(a1) normalizing the input face image with respect to the reference 

image; 

V - 

(a2) H estimating the reference image based shape information by 
15 performing a hierarchical, gradient-based optical flow algorithm on the 
normalized input face image and the reference image; 

(a3) compensating for an error value of the shape information 
estimated in step (a2) by performing linear decomposition and linear 
superposition on the estimated shape information based on shape 
20 eigenvectors stored in the image database in advance; 

(a4) transforming the shape of the input face image into the shape 
of the reference image using the shape information compensated in step 
(a3); 

(a5) determining the texture information about the input face 
25 image by performing linear decomposition and linear superposition on 
texture information about the image transformed in step (a4) based on 
texture eigenvectors stored in the image database in advance; and 

(a6) reflecting the results of steps (a4) and (a5) and repeating 
steps (a2) through (a5) until a predetermined condition is satisfied, 
30 thereby generating the shape information and the texture information 
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about the input face image based on the reference image. 

11. The method of claim 10, wherein step (a) further comprises 

the step of: 

*. 

5 (a0) generating the shape eigenvectors and the texture 

eigenvectors by performing a principal component analysis on a shape 
average, a texture average, a covariance of a shape difference, and a 
covariance of a texture difference which are obtained from reference 
image based shape information and texture information which is 

10 extracted from a plurality of model face images. 

12. The method of claim 9, wherein step (b) comprises the sub 
steps of: 

* - 

(b1) selecting a caricature image from among different styles of 
15 caricature images, which are stored in the image database and have the 
same shape as that of the reference image, according to a user control 
command; and 

(b2) synthesizing the caricature image selected in step (b1) or an 
image generated by performing a weighted summation of the selected 
20 caricature image and an image reflecting the extracted texture 
information with the shape information about the input face image, 
thereby generating a caricature image reflecting the shape information 
about the input face image. 

25 13. The method of claim 12, wherein step (b) further comprises 

the sub step of: 

(b3) when changing of the shape information is determined by a 
user control command, changing the shape information about the input 
face image according to a user control command controlling a change in 
30 the shape information and repeating steps (b1) and (b2). 

30 
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14. The method of claim 12, further comprising the step of: 

(c) replacing a part or the entire portion of the caricature image 
synthesized in step (b) with a sample image having a highest similarity to 

5 the synthesized caricature image among sample images stored in the 
image database. 

15. The method of claim 14, wherein the similarity is 
determined in accordance with a weighted sum of differences in shape 

10 information between the synthesized caricature image and the sample 
image and the sum of differences in texture information between the 
synthesized caricature image and the sample image, or a weighted sum 
of the sum of differences between eigenvector coefficients representing 
the shape information about the synthesized caricature image and 

is eigenvector coefficients representing the shape information about the 
sample image and the sum of differences between eigenvector 
coefficients representing the texture information about the synthesized 
caricature image and eigenvector coefficients representing the texture 
information about the sample image. 

20 

16. The method of claim 12 or 14, further comprising the step 

of: 

(d) adding an accessory image, which is selected from among 
accessory images stored in the image database according to a user 

25 control command, to the synthesized caricature image. 

17. The method of claim 16, wherein the position and size of 
the accessory image added in step (d) is determined in accordance with 
the shape information extracted in step (a) about the input face image. 

30 
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18. The method of claim 9, wherein step (b) comprises the sub 
steps of: 

(b1) selecting a makeup design image from among different styles 
of makeup design images, which are stored in the image database and 
5 have the same shape as that of the reference image, according to a user 
control command; 

(b2) synthesizing the makeup design image selected in step (b1) 
or an image generated by performing a weighted summation of the 
selected makeup design image and an image reflecting the extracted 
10 texture information with the shape information about the input face image, 
thereby generating a makeup design image reflecting the shape 
information about the input face image; and 

(b3) changing texture information about the reference image 
according to a user control command instructing revision of a makeup. 

15 

1 9. The method of claim 1 8, further comprising the step of: 

(c) adding an accessory image, which is selected from among 
accessory images stored in the image database according to a user 
control command, to the synthesized makeup design image. 

20 

20. The method of claim 19, wherein the position and size of 
the accessory image added in step (c) is determined in accordance with 
the shape information extracted in step (a) about the input face image. 
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